Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor and simplify logic in sdksCommon cmake script #3251

Merged
merged 1 commit into from
Jan 22, 2025

Conversation

SergeyRyabinin
Copy link
Contributor

@SergeyRyabinin SergeyRyabinin commented Jan 15, 2025

Issue #, if available:
cmake logic depends on c2j file list in tools/code-generation/api-descriptions
This does not make much sense anymore since

  1. generated clients are all located under generated/src source tree
  2. c2j file renaming logic is extracted and duplicated within the code generation script run_code_generation.py, so additional logic of datetime extraction, c2j filename renaming and etc is not actually required in the cmake script.

i.e. cmake can rely solely on the source tree

tested by building both full main and this branch and by comparing the cmake output and resulting build/install directories, i.e.

tree ~/sdk-install-main-clean/ > mainTree.log
tree ~/sdk-install-cmake-refactor/ > refactorTree.log
diff mainTree.log refactorTree.log

Description of changes:

Check all that applies:

  • Did a review by yourself.
  • Added proper tests to cover this PR. (If tests are not applicable, explain.)
  • Checked if this PR is a breaking (APIs have been changed) change.
  • Checked if this PR will not introduce cross-platform inconsistent behavior.
  • Checked if this PR would require a ReadMe/Wiki update.

Check which platforms you have built SDK on to verify the correctness of this PR.

  • Linux
  • Windows
  • Android
  • MacOS
  • IOS
  • Other Platforms

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@SergeyRyabinin SergeyRyabinin marked this pull request as ready for review January 16, 2025 17:46
@SergeyRyabinin SergeyRyabinin changed the title [draft] Refactor and simplify logic in sdksCommon cmake script Refactor and simplify logic in sdksCommon cmake script Jan 16, 2025
get_map_element(${PROJECT_NAME} TEMP_VAR ${SDK_DEPENDENCY_LIST})
get_flatmap_value(${PROJECT_NAME} TEMP_VAR ${SDK_DEPENDENCY_LIST})
if (TEMP_VAR)
string(REPLACE "," ";" TEMP_LIST_VAR ${TEMP_VAR})
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

one thing I see is because get_flatmap_value returns a cmake list, it already separates by semicolon, so we do not need this line I think. It doesn't hurt but will simplify even more on removal

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I saw this and even tried to get rid of this replacements as well, but
while it is indeed tries to return a semicolon-separated cmake list,
thanks to some cmake deep "magic", it is returned as a comma-separated list, and then it starts to fail in other places of the project in very strange ways (like when both comma and semicolon just disappear and the whole list is concatenated into a single word).

We can address it later (if it is even possible to fix)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting. Thanks for letting me know

LIST(APPEND C2J_SPECIAL_NAME_LIST "dynamodbstreams:streams.dynamodb")
set (svc "dynamodbstreams")
elseif("${svc}" MATCHES "\\.")
string(REPLACE "." ";" nameParts ${svc})
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These special name mapping is also captured in https://github.com/aws/aws-sdk-cpp/blob/main/tools/code-generation/smithy/codegen/smithy2c2j_service_map.json
Please just ensure the original naming is retained otherwise smoke tests wont build because they are built using the cmake library name which mimics the folder name
example: https://github.com/aws/aws-sdk-cpp/blob/main/generated/smoke-tests/apigateway/CMakeLists.txt#L4

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please read the PR description first, I've actually documented this:

  1. c2j file renaming logic is extracted and duplicated within the code generation script run_code_generation.py, so additional logic of datetime extraction, c2j filename renaming and etc is not actually required in the cmake script.

I've run an additional test today, just in case:

# Build main branch and install
# git branch: main
rm -rf buildMainClean && cmake -B buildMainClean -S . -DCMAKE_INSTALL_PREFIX=/home/srryabin/sdk-install-main -DENABLE_SMOKE_TESTS=ON -DCMAKE_BUILD_TYPE=Debug -DCMAKE_TOOLCHAIN_FILE=toolchains/clang-c++20.cmake &> configureMain.log && cmake --build buildMainClean -j 32 --config Debug --target install &> buildMain.log

# Build this branch and install
# git branch: sr/cmakeListClients
rm -rf buildRefactor && cmake -B buildRefactor -S . -DCMAKE_INSTALL_PREFIX=/home/srryabin/sdk-install-refactor -DENABLE_SMOKE_TESTS=ON -DCMAKE_BUILD_TYPE=Debug -DCMAKE_TOOLCHAIN_FILE=toolchains/clang-c++20.cmake &> configureRefactor.log && cmake --build buildRefactor -j 32 --config Debug --target install &> buildRefactor.log

# Compare the build output, the only difference is the output from the test testing race conditions on the logging (which is pretty okay)
tree buildRefactor > cmakerefac_tree.log
tree buildMainClean > cmakeclean_tree.log
diff cmakeclean_tree.log cmakerefac_tree.log

1c1
< buildMainClean
---
> buildRefactor
24739a24740
>     │   ├── aws_sdk_2025-01-21-18.log
24847c24848,24849
<     │   └── testRaceOnLogFile2025-01-21-17.log
---
>     │   ├── testRaceOnLogFile2025-01-21-17.log
>     │   └── testRaceOnLogFile2025-01-21-18.log
25487c25489
< 4748 directories, 20736 files
---
> 4748 directories, 20738 files

63 smoke tests app are built in both cases.

Smoke test for a legacy renamed service "apigateway" also builds and runs fine on my dev machine:

~/workspace/github/aws-sdk-cpp]
my-super-duper-dev-machine $ ./buildRefactor/generated/smoke-tests/apigateway/apigateway-smoke-tests
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from APIGatewaySmokeTestSuite
[ RUN      ] APIGatewaySmokeTestSuite.GetDomainNamesSuccess
[       OK ] APIGatewaySmokeTestSuite.GetDomainNamesSuccess (61 ms)
[----------] 1 test from APIGatewaySmokeTestSuite (61 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (96 ms total)
[  PASSED  ] 1 test.

-DBUILD_ONLY logic was also tested.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for verifying.

Copy link
Contributor

@sbera87 sbera87 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes look fine overall. Just a minor comment and request to verify if name change affects smoke tests.

@sbera87 sbera87 self-requested a review January 21, 2025 18:19
@SergeyRyabinin SergeyRyabinin merged commit 3e72626 into main Jan 22, 2025
5 checks passed
@SergeyRyabinin SergeyRyabinin deleted the sr/cmakeListClients branch January 22, 2025 17:10
sbera87 added a commit that referenced this pull request Jan 31, 2025
# This is the 1st commit message:

s3 code gen updates

# This is the commit message #2:

fixes

# This is the commit message #3:

updates

# This is the commit message #4:

updates

# This is the commit message #5:

compiling version

# This is the commit message #6:

updates

# This is the commit message #7:

test changes

# This is the commit message #8:

updates

# This is the commit message #9:

temp changes in s3client

# This is the commit message #10:

for debug

# This is the commit message #11:

Update API model

# This is the commit message #12:

corrects the dual-stack endpoint configuration for cognitoidp
Added DeleteContactFlowVersion API and the CAMPAIGN flow type
AWS IoT SiteWise now supports ingestion and querying of Null (all data types) and NaN (double type) values of bad or uncertain data quality. New partial error handling prevents data loss during ingestion. Enabled by default for new customers; existing customers can opt-in.
Documentation-only update to address doc errors
Documentation-only update: clarified the description of the shareDecaySeconds parameter of the FairsharePolicy data type, clarified the description of the priority parameter of the JobQueueDetail data type.
Added `DigitGroupingStyle` in ThousandsSeparator to allow grouping by `LAKH`( Indian Grouping system ) currency. Support LAKH and `CRORE` currency types in Column Formatting.
This release adds support for the topic attribute FifoThroughputScope for SNS FIFO topics. For details, see the documentation history in the Amazon Simple Notification Service Developer Guide.
Increasing entryPoint in SparkSubmit to accept longer script paths. New limit is 4kb.

# This is the commit message #13:

upgrade smithy version (#3263)


# This is the commit message #14:

Refactor and simplify logic in sdksCommon cmake script to not rely on c2j model (#3251)


# This is the commit message #15:

Update API model

# This is the commit message #16:

Adds multi-turn input support for an Agent node in an Amazon Bedrock Flow
Docs Update for timeout changes
Rename WorkSpaces Web to WorkSpaces Secure Browser
AWS Elemental MediaLive adds a new feature, ID3 segment tagging, in CMAF Ingest output groups. It allows customers to insert ID3 tags into every output segment, controlled by a newly added channel schedule action Id3SegmentTagging.

# This is the commit message #17:

Fix serialization for query xml

# This is the commit message #18:

Update API model

# This is the commit message #19:

Added "future" allocation type for future dated capacity reservation

# This is the commit message #20:

add stable order for smoke tests + restoring client codegen (#3265)


# This is the commit message #21:

add protocol test models (clients+tests) to the repo
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants